Products are always increasing in complexity and end users expect more features and functionalities from each subsequent versions and revisions of a specific product. It is not unusual for a design team to juggle with literally thousands of requirements during the course of a project, such as for automotive or telecommunication systems.
Some of these requirements evolve during the product development cycle and modifications to the design occur to comply with these modified requirements. Problems are also found during the design and testing phase, which also generate changes to the product itself. Feedback from the users is also incorporated into subsequent revisions of a product and through lab and field trials. Minute changes and customer specific modifications are often introduced to the first few batches of a product (or prototypes of this product) on an individual basis while the manufacturing process is still in tuning mode.
All these changes occur at various levels of the products and have diverse impacts on the performance and functionality, thus they impact product testing. Changes include: software changes, electronic modifications (parameter adjustments, parts replacement, component variations), firmware modifications (programmable logic, clock synthesizers programming, EEPROM updates), manufacturing data (dates, product batch codes), labeling and marking, mechanical adjustments (heat sinks, sheet metal modifications, fasteners), cosmetics changes, user documentation specifications and test plans, etc.
Maintaining a validation status for these systems, where each requirement can be verified with at least one or more test cases, is a daunting task in this ever changing environment. Various tools and processes exist in the field of computer aided product development, but the vast majority of these tools are targeted for pure software systems, with the abstraction of the hardware subsystems. These tools usually consider each copy of the same revision of the software to be a functional clone equivalent and are usually not designed to track test configurations. Thus, they are not optimized to provide good test repeatability in real life situations when the functionality (or performance) of the system as a whole is considered. Coverage report against changing specifications or a modified system is impacted as well.
In a project, it is highly desirable to maintain a structured set of requirements and to aggregate various attributes to these requirements to track project progress and completion. When a structured set of requirements exist, a complete test plan can be put together listing how each requirement is being verified with one or many test cases. Unfortunately, most computer aided testing tools are currently centered around test case management where various strategies are used to record and playback test cases in the form of scripts or automated actions performed on a target system. These tools put more emphasis on attempts to parameterize the test cases in order to cover various corner cases (with or without automatic parameter pairing and other reduction schemes), but do not usually clearly map test cases to requirements in order to compute requirement coverage with tests.
Even though good top-down architecture and design methodologies are well known, a surprisingly wide-spread technique has also found use in many projects where requirement verification has been forfeited, often because of the added burden of the change requests to the requirements, the design itself or the test cases. The technique involves using spreadsheets or check lists to track overall system performance with a sub-set (and often ad-hoc based) list of test cases. In this situation, it is very difficult to perform effective requirement tracking, to assess specification compliance or generate coverage reports for each prototype.
Improvements to overcome some or all of the above-noted deficiencies are therefore desired.